Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities.

Importance: Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective: To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants: Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions: Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures: Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results: Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance: This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Online Versus In-Person Screening, Brief Intervention, and Referral to Treatment Training in Pediatrics Residents.

BACKGROUND: Pediatricians underestimate the prevalence of substance misuse among children and adolescents and often fail to screen for and intervene in practice. The American Academy of Pediatrics recommends training in Screening, Brief Intervention, and Referral to Treatment (SBIRT), but training outcomes and skill acquisition are rarely assessed. OBJECTIVE: We compared the effects of online versus in-person SBIRT training on pediatrics residents' knowledge, attitudes, behaviors, and skills. METHODS: Forty pediatrics residents were randomized to receive either online or in-person training. Skills were assessed by pre- and posttraining standardized patient interviews that were coded for SBIRT-adherent and -nonadherent behaviors and global skills by 2 trained coders. Thirty-two residents also completed pre- and postsurveys of their substance use knowledge, attitudes, and behaviors (KABs). Two-way repeated measures multivariate analyses of variance (MANOVAs) and analyses of variance (ANOVAs) estimates were used to assess group differences in skill acquisition and KABs. RESULTS: Findings indicated that both groups demonstrated skill improvement from pre- to postassessment. Results indicated that both groups increased their knowledge, self-reported behaviors, confidence, and readiness with no significant between-group differences. Follow-up univariate analyses indicated that, while both groups increased their SBIRT-adherent skills, the online training group displayed more "undesirable" behaviors posttraining. CONCLUSIONS: The current study indicates that brief training, online or in-person, can increase pediatrics residents' SBIRT skills, knowledge, self-reported behaviors, confidence, and readiness. The findings further indicate that in-person training may have incremental benefit in teaching residents what not to do.

Validation of an Evidence-Based Medicine Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET).

BACKGROUND: Competence in evidence-based medicine (EBM) is an important clinical skill. Pediatrics residents are expected to acquire competence in EBM during their education, yet few validated tools exist to assess residents' EBM skills. OBJECTIVE: We sought to develop a reliable tool to evaluate residents' EBM skills in the critical appraisal of a research article, the development of a written EBM critically appraised topic (CAT) synopsis, and a presentation of the findings to colleagues. METHODS: Instrument development used a modified Delphi technique. We defined the skills to be assessed while reviewing (1) a written CAT synopsis and (2) a resident's EBM presentation. We defined skill levels for each item using the Dreyfus and Dreyfus model of skill development and created behavioral anchors using a frame-of-reference training technique to describe performance for each skill level. We evaluated the assessment instrument's psychometric properties, including internal consistency and interrater reliability. RESULTS: The EBM Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET) is composed of 14 items that assess residents' EBM and global presentation skills. Resident presentations (N  =  27) and the corresponding written CAT synopses were evaluated using the EBM C-PET. The EBM C-PET had excellent internal consistency (Cronbach α  =  0.94). Intraclass correlation coefficients were used to assess interrater reliability. Intraclass correlation coefficients for individual items ranged from 0.31 to 0.74; the average intraclass correlation coefficients for the 14 items was 0.67. CONCLUSIONS: We identified essential components of an assessment tool for an EBM CAT synopsis and presentation with excellent internal consistency and a good level of interrater reliability across 3 different institutions. The EBM C-PET is a reliable tool to document resident competence in higher-level EBM skills.

Is there a gap between program directors' expectations and residents' performance? Comparing predictions with outcomes for competence in patient care.

PURPOSE: To determine whether a gap exists between program directors' expectations for performance and residents' actual performance on learning objectives designed to address the Accreditation Council for Graduate Medical Education's (ACGME's) competencies. METHOD: In 2007-2008, the authors analyzed data on the performance of 40 pediatric residents across all levels of training who were assessed a total of 8,974 times during 2002-2005 on 35 learning objectives developed to address the ACGME patient care competency. Residents "passed" the objectives if assessments met the expected performance standards for each level of training that were set by pediatric program directors in a previous survey. To address concurrent validity, the authors also reviewed faculty evaluations of resident performance using a modified version of the American Board of Internal Medicine competency card. RESULTS: The mean pass rate (percentage of residents meeting or exceeding the predetermined standard) for the 35 learning objectives was 92% for first-year, 84% for second-year, and 72% for third-year residents. Actual performance met expected performance standards for all patient care learning objectives except those for procedural competence and some related to patient management. However, faculty competency card evaluations demonstrated that residents met or exceeded competence in patient care, suggesting that program directors set unrealistic standards for these few items. CONCLUSIONS: Program directors' expectations predicted residents' actual performance for the majority of patient care learning objectives. The authors offer lessons learned to inform the next iteration of performance standards for clinical competence.

Needle-free vaccine delivery.

The search for methods of vaccine delivery not requiring a needle and syringe has been accelerated by recent concerns regarding pandemic disease, bioterrorism, and disease eradication campaigns. Needle-free vaccine delivery could aid in these mass vaccinations by increasing ease and speed of delivery, and by offering improved safety and compliance, decreasing costs, and reducing pain associated with vaccinations. In this article, we summarize the rationale for delivery of needle-free vaccines and discuss several methods currently in use and under development, focusing on needle-free injection devices, transcutaneous immunization, and mucosal immunization. Jet injectors are needle-free devices that deliver liquid vaccine through a nozzle orifice and penetrate the skin with a high-speed narrow stream. They generate improved or equivalent immune responses compared with needle and syringe. Powder injection, a form of jet injection using vaccines in powder form, may obviate the need for the "cold chain." Transcutaneous immunization involves applying vaccine antigen and adjuvant to the skin, using a patch or "microneedles," and can induce both systemic and mucosal immunity. Mucosal immunization has thus far been focused on oral, nasal, and aerosol vaccines. Promising newer technologies in oral vaccination include using attenuated bacteria as vectors and transgenic plant "edible" vaccines. Improved knowledge regarding the immune system and its responses to vaccination continues to inform vaccine technologies for needle-free vaccine delivery.